Introducing Astrocytes on a Neuromorphic Processor: Synchronization, Local Plasticity and Edge of Chaos

While there is still a lot to learn about astrocytes and their neuromodulatory role in the spatial and temporal integration of neuronal activity, their introduction to neuromorphic hardware is timely, facilitating their computational exploration in basic science questions as well as their exploitation in real-world applications. Here, we present an astrocytic module that enables the development of a spiking Neuronal-Astrocytic Network (SNAN) into Intel's Loihi neuromorphic chip. The basis of the Loihi module is an end-to-end biophysically plausible compartmental model of an astrocyte that simulates the intracellular activity in response to the synaptic activity in space and time. To demonstrate the functional role of astrocytes in SNAN, we describe how an astrocyte may sense and induce activity-dependent neuronal synchronization, switch on and off spike-time-dependent plasticity (STDP) to introduce single-shot learning, and monitor the transition between ordered and chaotic activity at the synaptic space. Our module may serve as an extension for neuromorphic hardware, by either replicating or exploring the distinct computational roles that astrocytes have in forming biological intelligence.Comment: 9 pages, 7 figure

Empirical study on the efficiency of Spiking Neural Networks with axonal delays, and algorithm-hardware benchmarking

The role of axonal synaptic delays in the efficacy and performance of artificial neural networks has been largely unexplored. In step-based analog-valued neural network models (ANNs), the concept is almost absent. In their spiking neuroscience-inspired counterparts, there is hardly a systematic account of their effects on model performance in terms of accuracy and number of synaptic operations.This paper proposes a methodology for accounting for axonal delays in the training loop of deep Spiking Neural Networks (SNNs), intending to efficiently solve machine learning tasks on data with rich temporal dependencies. We then conduct an empirical study of the effects of axonal delays on model performance during inference for the Adding task, a benchmark for sequential regression, and for the Spiking Heidelberg Digits dataset (SHD), commonly used for evaluating event-driven models. Quantitative results on the SHD show that SNNs incorporating axonal delays instead of explicit recurrent synapses achieve state-of-the-art, over 90% test accuracy while needing less than half trainable synapses. Additionally, we estimate the required memory in terms of total parameters and energy consumption of accomodating such delay-trained models on a modern neuromorphic accelerator. These estimations are based on the number of synaptic operations and the reference GF-22nm FDX CMOS technology. As a result, we demonstrate that a reduced parameterization, which incorporates axonal delays, leads to approximately 90% energy and memory reduction in digital hardware implementations for a similar performance in the aforementioned task

Connecting Speech Encoder and Large Language Model for ASR

The impressive capability and versatility of large language models (LLMs) have aroused increasing attention in automatic speech recognition (ASR), with several pioneering studies attempting to build integrated ASR models by connecting a speech encoder with an LLM. This paper presents a comparative study of three commonly used structures as connectors, including fully connected layers, multi-head cross-attention, and Q-Former. Speech encoders from the Whisper model series as well as LLMs from the Vicuna model series with different model sizes were studied. Experiments were performed on the commonly used LibriSpeech, Common Voice, and GigaSpeech datasets, where the LLMs with Q-Formers demonstrated consistent and considerable word error rate (WER) reductions over LLMs with other connector structures. Q-Former-based LLMs can generalise well to out-of-domain datasets, where 12% relative WER reductions over the Whisper baseline ASR model were achieved on the Eval2000 test set without using any in-domain training data from Switchboard. Moreover, a novel segment-level Q-Former is proposed to enable LLMs to recognise speech segments with a duration exceeding the limitation of the encoders, which results in 17% relative WER reductions over other connector structures on 90-second-long speech data

Fine-grained Audio-Visual Joint Representations for Multimodal Large Language Models

Audio-visual large language models (LLM) have drawn significant attention, yet the fine-grained combination of both input streams is rather under-explored, which is challenging but necessary for LLMs to understand general video inputs. To this end, a fine-grained audio-visual joint representation (FAVOR) learning framework for multimodal LLMs is proposed in this paper, which extends a text-based LLM to simultaneously perceive speech and audio events in the audio input stream and images or videos in the visual input stream, at the frame level. To fuse the audio and visual feature streams into joint representations and to align the joint space with the LLM input embedding space, we propose a causal Q-Former structure with a causal attention module to enhance the capture of causal relations of the audio-visual frames across time. An audio-visual evaluation benchmark (AVEB) is also proposed which comprises six representative single-modal tasks with five cross-modal tasks reflecting audio-visual co-reasoning abilities. While achieving competitive single-modal performance on audio, speech and image tasks in AVEB, FAVOR achieved over 20% accuracy improvements on the video question-answering task when fine-grained information or temporal causal reasoning is required. FAVOR, in addition, demonstrated remarkable video comprehension and reasoning abilities on tasks that are unprecedented by other multimodal LLMs. An interactive demo of FAVOR is available at https://github.com/BriansIDP/AudioVisualLLM.git, and the training code and model checkpoints will be released soon

Open the box of digital neuromorphic processor: Towards effective algorithm-hardware co-design

Sparse and event-driven spiking neural network (SNN) algorithms are the ideal candidate solution for energy-efficient edge computing. Yet, with the growing complexity of SNN algorithms, it isn't easy to properly benchmark and optimize their computational cost without hardware in the loop. Although digital neuromorphic processors have been widely adopted to benchmark SNN algorithms, their black-box nature is problematic for algorithm-hardware co-optimization. In this work, we open the black box of the digital neuromorphic processor for algorithm designers by presenting the neuron processing instruction set and detailed energy consumption of the SENeCA neuromorphic architecture. For convenient benchmarking and optimization, we provide the energy cost of the essential neuromorphic components in SENeCA, including neuron models and learning rules. Moreover, we exploit the SENeCA's hierarchical memory and exhibit an advantage over existing neuromorphic processors. We show the energy efficiency of SNN algorithms for video processing and online learning, and demonstrate the potential of our work for optimizing algorithm designs. Overall, we present a practical approach to enable algorithm designers to accurately benchmark SNN algorithms and pave the way towards effective algorithm-hardware co-design

Rugged, Tunable Extended-Cavity Diode Laser

A rugged, tunable extended-cavity diode laser (ECDL) has been developed to satisfy stringent requirements for frequency stability, notably including low sensitivity to vibration. This laser is designed specifically for use in an atomic-clock experiment to be performed aboard the International Space Station (ISS). Lasers of similar design would be suitable for use in terrestrial laboratories engaged in atomic-clock and atomic-physics research

Structure and stability of arsenate adsorbed on α-Al2O3 single-crystal surfaces investigated using grazing-incidence EXAFS measurement and DFT calculation

Direct characterization of contaminants on single-crystal planes is required because the specific adsorption characteristics on different exposed crystal planes constitute their actual behavior at water–mineral interfaces in aquifers. Here, the structure and stability of arsenate on α-Al2O3 (0001) and (View the MathML source112¯0) surfaces were characterized by using a combination of grazing-incidence extended X-ray absorption fine structure (GI-EXAFS) spectra and periodic density functional theory (DFT) calculation. The combined results indicated that arsenate was mainly adsorbed as inner-sphere monodentate and bidentate complexes on both surfaces, but the orientational polar angles on the (0001) surface were commonly 10–20° greater than that on the (View the MathML source112¯0) surface. The DFT calculation showed that the large polar angle was more favorable for arsenate stabilized on the alumina surfaces. Based on the spectroscopic and computational data, the dominant bonding modes of arsenate on the two crystal planes of α-Al2O3 were identified as bidentate binuclear structures, and the (0001) surface displayed a stronger affinity toward arsenate

Verification of specific G-quadruplex structure by using a novel cyanine dye supramolecular assembly: II. The binding characterization with specific intramolecular G-quadruplex and the recognizing mechanism

The supramolecular assembly of a novel cyanine dye, 3,3′-di(3-sulfopropyl)-4,5,4′,5′-dibenzo-9-ethyl-thiacarbocyanine triethylammonium salt (ETC) was designed to verify specific intramolecular G-quadruplexes from duplex and single-strand DNAs. Spectral results have shown that ETC presented two major distinct signatures with specific intramolecular G-quadruplexes in vitro: (i) dramatic changes in the absorption spectra (including disappearance of absorption peak around 660 nm and appearance of independent new peak around 584 nm); (ii) ∼70 times enhancement of fluorescence signal at 600 nm. Furthermore, based on 1H-nuclear magnetic resonance and circular dichroism results, the preferring binding of ETC to specific intramolecular G-quadruplexes probably result from end-stacking, and the loop structure nearby also plays an important role